How do I get onto the island just south of the castle

Hint 1 of 2:

One if by land, two if by sea, three if by...